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SOME COMMON FLAWS IN EDUCATIONAL RESEARCH DESIGN WERE 
analyzed on the basis of a study by dr. GERALD SMITH OF 1,091 
UNSUCCESSFUL RESEARCH PROPOSALS. THE FOLLOWING INADEQUACIES 
ARE DISCUSSED— (1) THE LACK OF SPECIFICITY IN EVERY ASPECT 
OF THE research DESIGN, (2) THE NON-DELIMI TED PROBLEM, (3) 

THE LACK OF THEORETICAL FRAMEWORK FROM WHICH TO APPROACH THE 
PROBLEM, (4) THE LACK OF SIGNIFICANCE (USUALLY 
OVER-SIMPLIFICATION OF A COMPLEX PROBLEM), (5) THE FAILURE TO 
EXAMINE THOROUGHLY AND CRITICALLY PREVIOUS RESEARCH IN THE 
AREA, (6) THE PROVINCIALISM IN THE REVIEW OF RESEARCH 
(LIMITED SCOPE, GEOGRAPHY, SUBSTANCE, AND METHOD), (7) THE 
FAILURE TO INCLUDE REVIEW OF RELATED RESEARCH IN THE 
PROPOSAL, (6) THE FAILURE TO INCLUDE OBJECTIVES, HYPOTHESES, 
AND QUESTIONS OR INCLUSION OF BROAD OR VAGUE STATEMENTS OF 
OBJECTIVES, (9) INADEQUACIES IN SAMPLING (INCOMPLETENESS, 

LACK CF CLARITY IN SAMPLING PLAN, LACK OF WELL-DEFINED 
POPULATION) , (10) THE FAILURE TO DESCRIBE TECHNIQUES AND 

INSTRUMENTS TO BE USED IN THE RESEARCH, (11) INCOMPLETE 
DESCRIPTION OF THE STATISTICAL TREATMENT OF THE DATA, (12) 

THE USE OF UNSCIENTIFIC LANGUAGE OR OVERUSE OF TECHNICAL OR 
PSEUDO- technical LANGUAGE, AND (13) f(0 CONTROL FOR THE 
HAWTHORNE EFFECT. TEN CONCLUSIONS DRAWN FROM THE ANALYSIS OF 
research- DESIGN INADEQUACIES CONCERNING THE AREA OF 
EDUCATIONAL RESEARCH ARE LISTED. THIS PAPER WAS PRESENTED TO 
THE NATIONAL READING CONFERENCE (SAINT PETERSBURG, DECEMBER 
2, 1966). (LS) 
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FLAWS IN RESEARCH DESIGN 

RcGoarch t:o the scientist is a careful and systematic inquiry, usually 
requiring considerable time and using the best developed techniques. Research 
starts with a purpose, an intent to solve a clearly conceived problv..m. If the 
problem is to be solved, it is essential that the most appropriate research 
design be used. However, once a particular design is selected, it may contain 

The most serious flaw in a research design is lack of specificity. If 
soecificity is not evident in every aspect of a research design, the whole 
proposal is suspect because the reader cannot be sure v/nat the investigator 
intends to do. In fact, there- is room for considerable doubt that the investi- 
gator really knows what he intends to do. The reader must be sure what the pro- 
blem is, what the hypotheses^ state, and what statistical design the investigator 
plans to use. Both the reader and the investigator must know what is meant. 

The reader can be guided only by what is stated. 

During the several years I spent with the Bureau of Research of the U.S.O.E., 
several hundred research proposals came across my desk, approximately twenty per 
cent of which were approved. Every proposal which comies to the Bureau of Research 
is read carefully by several members of the Office staff. In addition proposals 
are read by research scholars representing every major area of research interest. 



Talk given at Annual Meeting of National Reading Conference, December 2, 1966, 
St. Petersburg, Florida. ^ 
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U.S. DEPARTMENT OF HEALTH, EDUCATION & WELFARE 
OFFICE OF EDUCATION 

THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROM THE 
PERSON OR 0R6ANIZATI0N ORIGINATING IT. POINTS OF VIEW OR OPINIONS 
STATED DO NOT NECESSARILY REPRESENT OFFICIAL OFFICE OF EDUCATION 
POSITION OR POLICY. 
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Proposrils which are not approved arc so rated generally because oi 
basic flaws, usually in design, which wake thew unacceptable as good res. -area. 
Some proposals arc approved with cernain provisions. This rricans tn.^t ^laws 
exist in the design, but in tne opinion oi tne evalUt.vOj. s >-nc i.ia.ws <-<_.** o~- 
corrocted. In these cases the researcli coordinator acquaints the i^v'est j.gc...or 
with the stipulated provisions and worlcs with hir.i to irr.prove the proposax so 
that a contract can be negotiatea. 

Cne of ity forrr.er colleagues. Dr, Gerald Sitith, nuw at Syracuse U..iv>-.- o- i-y , 
analysed 1,091 unsuccessful research proposals., ■ 416 of vdvich v?ere eventuaxiy 
eiirroinated because they did not follow the Cooperative Research lorn.at or nad 
been submitted previously. 

His original study is available in unpublished form at Columbia University 
and is titled Inadequacies in a Selected Sample of Educational Research ?ror..:sa: 



One of the most common inadequacies in research design is the way .he pro- 
blem is stated. The problem section of a research proposal snould outline and 
dclim.it the problem clearly.- An example of a non-deiimited prooiem is a ^jro- 
posal which sought to locate new sources of individuals who- could ne recruiL.cd 
into edu-cation as teachers. The problem section consisted of a series of ques- 
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1. How many liberal arts graduates can be recruitee into teaching ana 
how mariy of these are in rural or in urban areas? 

2. Is it possible to reorganize our schools and use our teachers more 

effectively? 

3. Can the problem be solved by using teacher aides? 

4. How many people have valid teaching certiticates and are not using 
them? Could they be induced to return to teaching? 
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If so, what refresher courses are needed? 

AH of these questions appeared in the probler. section of one proposal and 
serve as a prieie example of failure co delimit a problem. Several research 
projects would be needed to answer ail the questions presented. 

Smith found, in examining proposals submitted to the Cooperative Research 
Program during a three year period, that the most frequent inadequacy, which 
occurred 87 per cent of the. time in the problem section, was the lack of a 
theoretical frame work from which to approach the problem. Smith states that 
a full developed theory has (1) an explicit set of basic postulates or assump- 
tions, and (2) an equally explicit set of logically derived hypotheses. 

The problem of lack of significance also occured frequently-&6 per cent 
of the time; usually an atteiApt was made to oversimiplify complex problems or 
present them in such a way that it did not appear that the investigator was 
building upon previous research in the area. An example is this statement 
of a problem; " , , identification of factors accounting for bimodalities 
in data on student achievement at an institution. 

Once an individual selects a problem for research, he must familiarize 
himself with the previous research conducted in the area. Frequently, a 
research project builds upon results of past research, or it may replicate a 
completed project to determine whether the same results will be obtained, and 
if not, to find the reasons for the differences. Certainly no researcher would 
be so naive that he would fail to examine the body of research to &ee what 
previous work had been done in the area. Such examination should represent a 
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critical analysis of the research design as well as the results of the 
research, and should deraonstrate a relationship between past research and 
the proposal. This relationship should be deraonstrated by indicating the 
basic weakness, if any, of previous approaches, by indicating that the pre- 
sent study follows leads uncovered by past efforts, and by den;onstrating that 
the proposed study is significantly different from past efforts. A listing 
or summary of research with no description of its contributions to the field 
is of little value. 

Provincialism in a review of research may be another inadequacy. Smith's 
analysis revealed four types: provincialism of scope, of geography, of sub- 

stance, and of method, A proposal based on a single research study certainly 
is narrow in scope and placed unnecessary restraints on the investigator, for 
the rest of the related literature should provide something of value such as 
suggestions concerning sampling, instrumentation, theoretical fraraework or 
Qtha- aspects of the total research process. If all studies’ cited v;ere con- 
ducted at a single institution, one wonders if the investigator didn't want 
to bother to check another library. If all studies reviewed were carried on 
in a single state, one wonders if projects conducted in other states might not 
have provided, useful information. If the related research is concentrated on 
only a limited number of substantive aspects of a problem, the investigator 
does not have an adequate base for his research. Provincialism of method is 
obvious if the studies reviewed are conducted with the same instrument. Pro- 
vinciaij^T'm indicates that only a part of the total past research effort was 
considered and that more information which was relevant to the current problem 
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and ".aans of attacking it could have been used by the rasea.rcher. 

A more serious inadequacy in the research proposal is the failure cc 
include a reviev; of related research, Scnie proposals contain nothing but 
reviews of related literature. One assumes that an investigator examines 
the body of research before he outlines his proposal. The inclusion of 
ncn-resoarch references provides no information concerning rhe researcher's 
knov’ledge of the previous research. Although the non-research references 
may provide valuable information, the foundation for new research should be 
previous research. 

In citing research some investigators generalize without supporting 
evidence, or simply state that research has been conducted in the area, or 
amplify results of investigations without specifically citing them. Such 
proposals are usually not approved for support. The proposal must contain 
a firm and specifically stated base in previous research. The proposal should 
also contain pertinent references to related literature. 

Smith found that three per cent of the proposals he examined included 
no objectives, hypotheses, or questions. An additional 63 per cent were 
inadequate in the clarity with which the objectives, hypotheses, or questions 
v;ere presented. The objectives and hypotheses may be stated too broadly for 
research purposes; for example: 

The broad hypothesis is that a procedure can be followed which will lead 
to the initial formulation, revision, and final development of a broadly 
conceived theory of education based upon psychological and other relevant 
research findings to date. 
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Probably cho writer had something in mind when 
uc it is di££icult for the reader to determine just wh 



he made that statement. 
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Perha'^^3 the writer has w’crked so incimateiy w^ith his proposal that he assumes 
that his reader knows as much about the problem as he does. 

In a well written proposal not only are the objectives, hypotheses, and 
questions cit^rly stated, but the technical concepts to be employed also are 
defined with clarity. 

If the relationship between the theoretical framcv;ork and the objectives, 

hypotheses, or questions is to be adequate, the froj:;ework and theory must be 

1 

developed in the statement of the problem. 

When one thinks of flaws in design, the first thing that comes to mind is 
the procedure since it is so intimately a part of the design. In determining 
the population to be used in a study, the research must employ appropriate sam* 
pling techniques. Coirimon inadequacies in sampling include incompleteness, a 
lack of clarity in the sampling plan, and lack of a v;ell defined population. 

Occasionally a researcher will use a sample which is based not upon the 
purposes of the study but rather upon the proximity of a particular group of 
subjects. Perhaps he may use an intact group, a group which existed before 
the study began. He must demonstrate clearly that such a group is adequate 
for the study he proposes. 
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Convenience and econoray are legitiiiiate criteria to use in selecting 



only. criteria^ 


the sample 


must be 
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popul^.ticn 
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the sar.iple is representative. A sarr.plo should be representative of a 
particular^ iinite population, and this population should be described 
in the proposal. 

Another inadequacy in proposals concerns instrumentation. Occasionally 
the techniques and/or instruments are not described clearly ana completely; 
if the investigator simply states that he v;ili use a reading and achievement 
test but fails to state which test, the. read.er cannot determine whether the 
instrumentation is adequate to gather the necessary data. If the instrument 
will provide reasonably reliable data to answer the questions raised in the 
hypotheses, it is an appropriate instrument to use. The types of instruments 
vary with the purpose of the study, a questionnaire or a check list might be 
appropriate in one proposal and totally inappropriate in another. 

Smith found that 17 per cent ‘of the proposals he examined did not pre- 
sent a complete description of the statistical treatment of the data. Unless 
a complete description of the statistical method is given, the reader cannot 
discover v;hat the investigator plans to do. 

Another area of inadequacy is found in the language used for communica- 
tion. Reading published research is often difficult for the layman if a great 
deal or statistical terminology is used. However, research proposals are 



er|c 









-V 



0 



8 



wriiiticn to be bead critically by people with the vocabulary of 

research desien and statistics* ScnvCtiues the writer may er.iploy too 
much unscientitic language; sometimes he organizes his proposal poorly, 
ana less often he may produce a proposal which lacks conciseness and suf- 
ficient cetaii ana is difficult to read because the author overuses tech- 
nical or pseudo-technical language. 

A relatively nev; term now in popular use is "Hav;thorne Effect". It 
refers to any fault in research by which one group of subjects either 
receives or believes they are receiving special consideration during the 
course of the study v^hich would not have been given to a control group. 

The chance of a Hawthorne effect must be controlled or the. results of a 
study may be invalidated. ‘ 

Smith drew a number of conclusions from his examination of the re- 
search proposals. However, as a good rese-areher does, he warned that they 
are true for educational research only as the entire field of educational 
research is reflected in his sample. 

1. The state of educational x’esearch i^ at a relatively low level of 
:sophistication. 

2. There is little agreement about what constitutes research in education. 

3. Much of educational research is focused on relatively simple, straight 
forward, "surface" problems. 
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Taore is little emphasis upon theory in educational research. 
Researchers have difficulty in differentiating betweun research 



anc practice in education. 



dducaticnai researchers are making only minimal use of the prcvioi 
researen which has bek.-.n cone on similar problems. 

There is little agreement about what a review of related research 
is supposed to accemolisn. 



b. aoseUi. chers are paying too little attention to the problaro of good 
cemmuni cation . 

9. lias wily conceived ideas are translated into proposals without a 
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sufficient period of germination. 

^ne lacu or detail *is pernaps the one single; miOst frequently 
occurring inadequacy in educational research proposals. 

Smii_n's list of inadequacies or flaws in design of educational 
research may sound too much like an indictment of all efforts, but they 
should not be thought of in those terms. They represent a consensus ef 
reasons wny more than 1,0C0 research proposals were judged not worthy of 
Federal support. 

^iany Oj. tnese proposals started out with worthwhile questions or 
icec*s. It is tragic that good ideas are not translated into good research 
proposals. The purpose of educational research is to discover truth. A 

rc;seai.ch eirort that is successful yields a firm base for quality education. 

Quality education is worth the effort required to design a good research nro 
posal. 
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